Recently, various techniques and products for obtaining a 3D image from a recognized object have been developed. For example, a TOF (Time Of Flight) system is to obtain a 3D image from a distance or depth between a camera and a recognized object, which is measured using a temporal difference between a light emission time for irradiating light to the recognized object and a light receiving time of light reflected from the recognized object. Also, a structure light system is to obtain a 3D image from a depth of a recognized object, which is measured by emitting patterned infrared structured light to the recognized object and analyzing a pattern of infrared rays received from the recognized object.
In this regard, although two or more visible light images may be used to obtain 3D depth information, a system, which uses the visible light images together with infrared ray images as an active light source, has been used recently. Also, in this system, it is general that a separate sensor (RGB sensor) for taking visible light is provided together with an infrared ray sensor (IR sensor). Furthermore, a camera sensor structure for obtaining visible light images and infrared ray images from one RGB-IR sensor by modifying one of sensor pixels for taking visible light to a pixel for obtaining infrared rays has been studied.
FIG. 1 is a diagram illustrating a pixel arrangement structure of an RGB-IR sensor used for obtaining a 3D image in accordance with the related art. Generally, the RGB-IR sensor obtains color images of a visible light area through R(Red), G (Green) and B (Blue) pixels and also obtain infrared images through infrared ray (IR) pixels. That is, a single RGB-IR sensor is configured by combination of R, G, B and IR pixel arrangement.
In this regard, FIG. 1 is an example illustrating a pixel arrangement order of an RGB-IR sensor according to the related art. Generally, the RGB-IR sensor structure is configured in such a manner that one G pixel is modified into IR pixel in a general RGB sensor (which is widely known as a ‘Bayer sensor structure’) that includes one R pixel, one B pixel and two G pixels.
In more detail, it is noted from the pixel arrangement structure of the related art RGB-IR sensor of FIG. 1 that a pixel basic unit structure 101 including R, G, B and IR pixels one by one is repeated at equivalent intervals. Therefore, all IR pixels within the RGB-IR sensor are arranged at equivalent intervals. Likewise, the other color components (R, G and B) are arranged at equivalent intervals. This structure will be referred to as a network or chessboard structure.
FIG. 2 is a diagram illustrating an interpolation method of IR pixels by using the related art RGB-IR sensor of FIG. 1. In this regard, FIG. 2 is a reference diagram illustrating how interpolation is performed in a specific pixel location by enlarging neighboring IR pixels 111, 112, 113, 114, 115, 116, 117, 118 and 119 around the IR pixel 115 of FIG. 1.
First of all, a general interpolation method between pixels will be described as follows. In order to obtain RGB color images and IR images, various interpolation methods are generally applied to components obtained by each pixel. For example, a demosaicing method is widely used as the interpolation method. The demosaicing method is a kind of color filter interpolation method, and means an image processing algorithm for recovering a full color value of all pixels in pixel arrangement. For example, various interpolation methods suitable for a corresponding purpose, such as sum of average weighted values and sum of edge based weighted values, are applied to the demosaicing method. Therefore, each pixel may be recovered to have all components (R, G, B, IR) through the demosaicing method.
Referring to FIG. 2 again, for example, a case where IR interpolation is to be performed in pixels 121, 122, 123, 124, 125, 126, 127 and 128, located among IR pixels 111 to 119 will be described. It is noted that two kinds of pixel interpolation methods are applied to FIG. 2. For example, in case of the pixel locations 122, 124, 126 and 128, IR pixels exist near the corresponding pixels, whereby interpolation may be performed using the corresponding IR pixels only. For example, in the pixel location 122, interpolation is performed using two IR pixels 112 and 115. Likewise, in the pixel location 124, interpolation is performed using two IR pixels 116 and 115, in the pixel location 126, interpolation is performed using two IR pixels 118 and 115, and in the pixel location 128, interpolation is performed using two IR pixels 114 and 115. In particular, the IR pixels 112, 114, 115, 116 and 118 adjacent to the interpolation location pixels 122, 124, 126 and 128 are tight coupled with the corresponding interpolation location pixels 122, 124, 126 and 128 are marked with solid lines as shown. The tight coupled IR pixels are characterized in that they are adjacent to a shortest distance from the pixel location where interpolation is to be performed.
On the other hand, in case of the pixel locations 121, 123, 125 and 127, IR pixels do not exist within the shortest distance adjacent to the corresponding pixels, whereby interpolation is performed using four IR pixels surrounding the pixel locations in a diagonal direction. For example, in the pixel location 121, interpolation is performed using four IR pixels 111, 112, 114 and 115. Likewise, in the pixel location 123, interpolation is performed using four IR pixels 112, 113, 115 and 116, in the pixel location 125, interpolation is performed using four IR pixels 115, 116, 118 and 119, and in the pixel location 127, interpolation is performed using four IR pixels 114, 115, 117 and 118.
In this regard, the IR pixels which are not adjacent from the interpolation location within the shortest distance are loose coupled with the corresponding pixel locations 121, 123, 125 and 127 and are marked with dotted lines as shown. That is, in the pixel locations 121, 123, 125 and 127, the tight coupled IR pixels within the shortest distance do not exist. Therefore, if interpolation is performed using the loose coupled IR pixels only, a problem occurs in that interpolation accuracy and efficiency are more deteriorated than the case where at least one or more tight coupled IR pixels exist.